1
Reporting and Learning from Health IT-Related Events
Toward Safer Healthcare
Session 166, Wednesday, February 13, 2019
Yang Gong, MD, PhD, UTHealth
2
Yang Gong, MD, PhD
Has no real or apparent conflicts of interest to report.
Conflict of Interest
3
• Introduction (~5 minutes)
– Moderator
– Speaker
• Presentation (~45 minutes)
– Scan QD code for details
• Q&A (~10 minutes)
• Extended discussion (Posterior to the session)
– Twitter @gngyng
– LinkedIn GONGYANG@gmail.com
Agenda
4
• State the benefits and risks of HIT for patient
safety/healthcare quality improvement
• Identify the barriers of reporting and analyzing HIT events
and challenges for turning HIT event reports into actionable
knowledge
• Discuss how data representation and knowledge
management in FDA MAUDE incident reports can facilitate
quality improvement towards a better and safer healthcare
system
Learning Objectives
5
Patient Safety
- Pressures and Incentives
Medical error. (Makary & Daniel, 2016)
Leveraging patient safety research: Fifteen-year
efforts since “To Err Is Human” (Liang, Miao, Kang… &
Gong, submitted manuscript, 2018)
6
Individual
Reports
Individual
Reports
• Clinicians
• Patients
Aggregated
Data
Aggregated
Data
• Hospitals
• PSO
AnalysisAnalysis
• Hospitals
• PSO
Actionable
Knowledge
Actionable
Knowledge
• Clinicians
• Hospitals
Resolving
Problems
Resolving
Problems
• Clinicians
• Hospitals
Improving Safety through Reporting
• Patient Safety Event (PSE)
Reporting Initiatives
• To Err is Human (2000)
• Patient Safety and Quality
Improvement Act (2005)
• Patient Safety Organizations (PSO)
• Why Reporting
• Learn from lessons
• Reporting Mechanism
States working with PSO by 2017
7
Perceived Barriers
• Lack of instructions and training
• Unsatisfactory usability of
classifications/taxonomies
– Lengthy reporting forms
• Time-consuming
– Competing with other priorities
• Lack of motives
– No feedback
– Observed event seemed “trivial”
» A trivial tip --> a large ‘iceberg’ under water
8
Challenges in Event Reporting
• Structured data vs. narratives
• Structured data: standardized but limited representations
• Narratives: flexible, content-rich, causal and temporal
information
• Complexity of medication error reports
8
Multi-Stage
Multi-Personnel
Multi-Factor
Reporting
Analysis
Actions
Learning
Barriers
9
Problem in Reporting
– Low Reporting Quality
Data Type
Ideal
ity
Reality
Structured data
Easy to
interpret
Rarely
answered
Unstructured data
Supplementary Info.
Essential Info.
10
Problem in Reporting Errors
- Lacking Analysis Tools
• A New Report: “RN removed 100mcg fentanyl from the
omnicell in endo room 2 documented on the anesthesia
sheet that he gave 25mcg, no waste recorded. 75mcg
fentanly unaccounted for.”
Historical Reports & Solutions
Reported, so what?
11
• Institute of Medicine (IOM)
– HIT plays an imperative role in saving healthcare cost, improving patient outcomes,
decreasing occurrence of medication errors, and refining healthcare process measures
across diverse settings
• Agency for Healthcare Research and Quality (AHRQ)
– use of information and communication technology in healthcare to support the delivery of
patient or population care or to support patient self-management
– Synonym
• electronic health records (EHR) and
• EHR components
– computerized provider order entry (CPOE) or
– clinical decision support system (CDSS)
– Generalized Health IT also includes
• administrative or practice management systems
• automated dispensing systems
• laboratory information systems, and
• diagnostic imaging systems
• Nowadays, Health IT has become an integral part of healthcare and has been widely applied to
collect, transmit, process, display, and store patient data
What is Health IT
12
• Listed in the top 10 technology-related hazards
– new uncertainties and risks for patient safety
– disrupting established work patterns
– creating new risks in practice, and
– encouraging workarounds
• The adoption of EHR has revealed potential safety
implications related to
– EHR design, implementation, and use related to
• technological features of EHR
• users and workflows
• organizations, rules and regulations
Challenges of Health IT
13
• Understand and manage the risks
– Sociotechnical context
– Sittig and Singh 8-dimensional sociotechnical model
• Tool in patient safety studies
• Complexities of technology
• Users in workflow and external or organizational policies
• Health IT in event reporting
– Citied as one of contributing factors in reporting systems
– No health IT exclusive sources to patient safety studies
– AHRQ Common Formats (CFs): Common definitions and
reporting formats (QD code linked)
• Hospitals, Community pharmacies, Nursing homes
Meeting the Challenges
14
Goal
• Develop a user-centered, knowledge-
based reporting and learning system
– Help healthcare practitioners better report events
– Connect with relevant reports
– Learn how to address causes of errors
– Improve the behavior at work
15
Method –Filter on Structured Data
• 4,947,220 reports from
MAUDE 2008-2016
• Keyword searching on
Generic Name and
Manufacturer Name
• Identify HIT-related events
from sample reports through
domain expert review
• 6 inclusion criteria
• 4 exclusion criteria
• Assess reviewer
consistency by Cohen’s
kappa
Workflow of reviewing a report from FDA MAUDE database
16
Method
– Classifier on Unstructured Data
• Term frequency (TF) – inverse document frequency (IDF)
• Biterm topic model
• Extract the semantic themes (topics) from a corpus of short
documents
• Classifiers
• Random forest, Logistic regression, Naïve Bayes, SVM, J48, JRip
• Gold standard:
• HIT-related and non-HIT event reports identified by domain experts
17
Result
– Keyword-based HIT Event Filter
Generic Names Manufacturer Names Total
Inclusion Keywords 94 38 132
Exclusion Keywords 21 0 21
Year
Raw reports Filtered reports
Reports
HCFA/Manufactur
er/Distributor
Reports
HCFA/Manufactur
er/Distributor
2008
145,598
9,148
1,817
146
2009
201,996
9,906
2,640
214
2010
327,961
10,792
3,434
316
2011
414,083
12,597
2,371
307
2012
520,043
12,952
4,825
308
2013
636,145
12,516
3,551
313
2014
867,451
12,927
4,338
380
2015
965,240
15,762
17,963
384
2016
868,703
15,023
4,685
408
Sum
4,947,220
45,624
• Inclusion & exclusion keywords
• Raw and filtered reports of 2008-2016 MAUDE database
18
Result
– Contributing Factor Distribution Analysis
19
Result
- FDA Medical Device Event Reports
• FDA Manufacturer and User Facility
Device Experience (MAUDE) database
• ~ 6,000,000 events involving medical
devices
• structured & unstructured data fields
• 0.46% ~0.69% are HIT-related events
– Up to 50,000 HIT-related events
(QD code linked)
Trend of MAUDE reports (bars) and MAUDE
related publications (line) since 2000
Filter on Structured
data
Classifier on
Unstructured data
Raw Data
HIT
20
Result
– An Integrated Model of HIT event Identification
0.4~0.9% 50% 97%
Proportion of
HIT events
Filter on
Structured
data
Classifier on
Unstructured
data
Raw Data HIT
Trade off between precision and recall
Grow the HIT database
21
Underway
– Prototype An Integrated Reporting and Shared
Learning System in Healthcare Community
22
Predictive Text Entry
• To support reporting
– Cueing list, auto-
suggestion
• By two-group
randomized test
– Improved text
generation
– Improved data
consistency and
quality
Entered and
tagged-in text
Initial letters
of input
Auto-suggestion:
matched text
entry hits
(# of hits <=10)
Narrative data entry field equipped with text prediction functions
E
F
G
C
B
Main component lists multiple-choice questions in slide-in mode
Cueing list that reminds
of the content or content
categories of reportable data
A
D
C
Structured Data Entry – 13 MCQs and four of them have narrative fields as illustrated as the part B
Unstructured Data Entry – One narrative comment field
C: Cueing List
aids in data entry of
specified single-text
field
(B) in the structured
question, or
comment field
G: Auto-suggestion
Suggesting the words, phrases
and sentence in the context to
describe the event details
23
Managing PSE Knowledge
• Ontology
– Interoperability among
• home-grown systems
• patient safety organization
(PSO) systems
– Data integration
• Organizing prevailing
classifications
– Decision making
24
Data Source & Annotation
• One year data from a PSO
institute (2016)
• 2,576 medication error
reports (including
adverse drug reaction,
ADR)
• Incidents & near misses
• Unstructured data
• Manual annotating
• Two patient safety
domain experts with
pharmacy or clinical
backgrounds
• Followed the NCC
MERP taxonomy for
medication errors, plus
ADR
Error
Stages
(6)
Error
Stages
(6)
Error
Types
(8)
Error
Types
(8)
Error
Causes
(5)
Error
Causes
(5)
Administering
Dispensing
Med. Rec.
Monitoring
Ordering
Transcribing
ADR
Billing Issue
Missing Dose
Wrong Admin.
Wrong Document.
Wrong Dose
Wrong Drug
Wrong Time
Devices (HIT)
Information Deficit
Pathophysiological Factor
Performance Deficit
Others
PSO Reports
25
Pipeline
– Identifying Three Key Factors
Remove punctuation
Remove number
Rainbow stop word
N-grams tokenizer
TF-IDF
26
Pipeline
– Similarity Measurement
Raw Reports
Similarity
Calculation
Grouped Reports
Labels:
• Originating stage
• Type
• Cause
27
Data Elements in Medication Error
28
Leveraging Patient Safety
• Information models act as the core
• Knowledge base plays an central role in the knowledge support
• Transforming reports into actionable knowledge
29
Knowledge Support
• Identify similar cases based on query
– Web M&M (PSNet)
– Patient Safety Organization (PSO) data
– Data from home-grown system
• Provide solutions and suggestions
30
Prototype
31
Innovative Design
Current Frames
• Reports are stored entry by entry
• Reporters learn nothing
• No feedback for systems
Proposed Frames
• Reports are annotated on the same feature tree
• Provide solutions for reporters
• The system can learn from user feedback and preferences
v
Feedback / Preferences
Knowledge Support (e.g., Solutions)
32
Identifying Relevant Cases
33
Exploring Event Connections
PSE Space
Topic Space
?
Topic Model
34
Developing a PSE Knowledge Base
Topics
Reports
Solutions
A PSE Knowledge Base
35
Acknowledgement
• Funding
– Agency for Healthcare Research & Quality, 1R01HS022895
– UTHealth Innovation in Cancer Prevention Research Training Program
(Cancer Prevention and Research Institute of Texas grant RP#160015)
– University of Texas System Grants Program #156374
• Current lab
members
– Hong Kang, PhD
– Pei-Yin Yang, MS
Alumni
Chen Liang, PhD
Ju Wang, PhD
Sicheng Zhou, MS
Bin Yao, MD, MS
Qi Miao, BS
Hsing-yi Song, MD, MS
Xinshuo Wu, MD, MS
Swananda Pandit, MS
Lei Hua, PhD
James Richardson, MS
Zhijian Luan, MS
Yanyan Shen, MHA
Rajitha Gopidi, MHA
Dan Wang, PhD
Mathew Koelling, MHA
CPRIT Summer
intern
Cindy Songting Wu
Frank Wang
Elisa Ali
Melanie Klock
Ethan Wang
Collaborators
Jing Wang, PhD, RN
Nnaemeka Okafor, MD, MS
Hua Xu, PhD
Tina Hilmas RN, BSN
Becky Miller MHA
Amy Vogelsmeier, PhD, RN
